<!-- HTML header for doxygen 1.8.6-->
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta http-equiv="Content-Type" content="text/xhtml;charset=UTF-8"/>
<meta http-equiv="X-UA-Compatible" content="IE=9"/>
<meta name="generator" content="Doxygen 1.8.13"/>
<title>OpenCV: Conversion of PyTorch Classification Models and Launch with OpenCV C++</title>
<link href="../../opencv.ico" rel="shortcut icon" type="image/x-icon" />
<link href="../../tabs.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../jquery.js"></script>
<script type="text/javascript" src="../../dynsections.js"></script>
<script type="text/javascript" src="../../tutorial-utils.js"></script>
<link href="../../search/search.css" rel="stylesheet" type="text/css"/>
<script type="text/javascript" src="../../search/searchdata.js"></script>
<script type="text/javascript" src="../../search/search.js"></script>
<script type="text/x-mathjax-config">
  MathJax.Hub.Config({
    extensions: ["tex2jax.js", "TeX/AMSmath.js", "TeX/AMSsymbols.js"],
    jax: ["input/TeX","output/HTML-CSS"],
});
//<![CDATA[
MathJax.Hub.Config(
{
  TeX: {
      Macros: {
          matTT: [ "\\[ \\left|\\begin{array}{ccc} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{array}\\right| \\]", 9],
          fork: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ \\end{array} \\right.", 4],
          forkthree: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ \\end{array} \\right.", 6],
          forkfour: ["\\left\\{ \\begin{array}{l l} #1 & \\mbox{#2}\\\\ #3 & \\mbox{#4}\\\\ #5 & \\mbox{#6}\\\\ #7 & \\mbox{#8}\\\\ \\end{array} \\right.", 8],
          vecthree: ["\\begin{bmatrix} #1\\\\ #2\\\\ #3 \\end{bmatrix}", 3],
          vecthreethree: ["\\begin{bmatrix} #1 & #2 & #3\\\\ #4 & #5 & #6\\\\ #7 & #8 & #9 \\end{bmatrix}", 9],
          cameramatrix: ["#1 = \\begin{bmatrix} f_x & 0 & c_x\\\\ 0 & f_y & c_y\\\\ 0 & 0 & 1 \\end{bmatrix}", 1],
          distcoeffs: ["(k_1, k_2, p_1, p_2[, k_3[, k_4, k_5, k_6 [, s_1, s_2, s_3, s_4[, \\tau_x, \\tau_y]]]]) \\text{ of 4, 5, 8, 12 or 14 elements}"],
          distcoeffsfisheye: ["(k_1, k_2, k_3, k_4)"],
          hdotsfor: ["\\dots", 1],
          mathbbm: ["\\mathbb{#1}", 1],
          bordermatrix: ["\\matrix{#1}", 1]
      }
  }
}
);
//]]>
</script><script type="text/javascript" src="https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.0/MathJax.js"></script>
<link href="../../doxygen.css" rel="stylesheet" type="text/css" />
<link href="../../stylesheet.css" rel="stylesheet" type="text/css"/>
</head>
<body>
<div id="top"><!-- do not remove this div, it is closed by doxygen! -->
<div id="titlearea">
<!--#include virtual="/google-search.html"-->
<table cellspacing="0" cellpadding="0">
 <tbody>
 <tr style="height: 56px;">
  <td id="projectlogo"><img alt="Logo" src="../../opencv-logo-small.png"/></td>
  <td style="padding-left: 0.5em;">
   <div id="projectname">OpenCV
   &#160;<span id="projectnumber">4.5.2</span>
   </div>
   <div id="projectbrief">Open Source Computer Vision</div>
  </td>
 </tr>
 </tbody>
</table>
</div>
<!-- end header part -->
<!-- Generated by Doxygen 1.8.13 -->
<script type="text/javascript">
var searchBox = new SearchBox("searchBox", "../../search",false,'Search');
</script>
<script type="text/javascript" src="../../menudata.js"></script>
<script type="text/javascript" src="../../menu.js"></script>
<script type="text/javascript">
$(function() {
  initMenu('../../',true,false,'search.php','Search');
  $(document).ready(function() { init_search(); });
});
</script>
<div id="main-nav"></div>
<!-- window showing the filter options -->
<div id="MSearchSelectWindow"
     onmouseover="return searchBox.OnSearchSelectShow()"
     onmouseout="return searchBox.OnSearchSelectHide()"
     onkeydown="return searchBox.OnSearchSelectKey(event)">
</div>

<!-- iframe showing the search results (closed by default) -->
<div id="MSearchResultsWindow">
<iframe src="javascript:void(0)" frameborder="0" 
        name="MSearchResults" id="MSearchResults">
</iframe>
</div>

<div id="nav-path" class="navpath">
  <ul>
<li class="navelem"><a class="el" href="../../d9/df8/tutorial_root.html">OpenCV Tutorials</a></li><li class="navelem"><a class="el" href="../../d2/d58/tutorial_table_of_content_dnn.html">Deep Neural Networks (dnn module)</a></li>  </ul>
</div>
</div><!-- top -->
<div class="header">
  <div class="headertitle">
<div class="title">Conversion of PyTorch Classification Models and Launch with OpenCV C++ </div>  </div>
</div><!--header-->
<div class="contents">
<div class="textblock"><p><b>Prev Tutorial:</b> <a class="el" href="../../dc/d70/pytorch_cls_tutorial_dnn_conversion.html">Conversion of PyTorch Classification Models and Launch with OpenCV Python</a></p>
<table class="doxtable">
<tr>
<th align="right"></th><th align="left"></th></tr>
<tr>
<td align="right">Original author </td><td align="left">Anastasia Murzova </td></tr>
<tr>
<td align="right">Compatibility </td><td align="left">OpenCV &gt;= 4.5 </td></tr>
</table>
<h2>Goals</h2>
<p>In this tutorial you will learn how to:</p><ul>
<li>convert PyTorch classification models into ONNX format</li>
<li>run converted PyTorch model with OpenCV C/C++ API</li>
<li>provide model inference</li>
</ul>
<p>We will explore the above-listed points by the example of ResNet-50 architecture.</p>
<h2>Introduction</h2>
<p>Let's briefly view the key concepts involved in the pipeline of PyTorch models transition with OpenCV API. The initial step in conversion of PyTorch models into <a class="el" href="../../db/d30/classcv_1_1dnn_1_1Net.html" title="This class allows to create and manipulate comprehensive artificial neural networks. ">cv::dnn::Net</a> is model transferring into <a href="https://onnx.ai/about.html">ONNX</a> format. ONNX aims at the interchangeability of the neural networks between various frameworks. There is a built-in function in PyTorch for ONNX conversion: <a href="https://pytorch.org/docs/stable/onnx.html#torch.onnx.export"><code>torch.onnx.export</code></a>. Further the obtained <code>.onnx</code> model is passed into <a class="el" href="../../d6/d0f/group__dnn.html#ga7faea56041d10c71dbbd6746ca854197" title="Reads a network model ONNX. ">cv::dnn::readNetFromONNX</a> or <a class="el" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422" title="Read deep learning network represented in one of the supported formats. ">cv::dnn::readNet</a>.</p>
<h2>Requirements</h2>
<p>To be able to experiment with the below code you will need to install a set of libraries. We will use a virtual environment with python3.7+ for this:</p>
<div class="fragment"><div class="line">virtualenv -p /usr/bin/python3.7 &lt;env_dir_path&gt;</div><div class="line">source &lt;env_dir_path&gt;/bin/activate</div></div><!-- fragment --><p>For OpenCV-Python building from source, follow the corresponding instructions from the <a class="el" href="../../da/df6/tutorial_py_table_of_contents_setup.html">Introduction to OpenCV</a>.</p>
<p>Before you start the installation of the libraries, you can customize the <a href="https://github.com/opencv/opencv/tree/master/samples/dnn/dnn_model_runner/dnn_conversion/requirements.txt">requirements.txt</a>, excluding or including (for example, <code>opencv-python</code>) some dependencies. The below line initiates requirements installation into the previously activated virtual environment:</p>
<div class="fragment"><div class="line">pip install -r requirements.txt</div></div><!-- fragment --><h2>Practice</h2>
<p>In this part we are going to cover the following points:</p><ol type="1">
<li>create a classification model conversion pipeline</li>
<li>provide the inference, process prediction results</li>
</ol>
<h3>Model Conversion Pipeline</h3>
<p>The code in this subchapter is located in the <code>samples/dnn/dnn_model_runner</code> module and can be executed with the line:</p>
<div class="fragment"><div class="line">python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_resnet50_onnx</div></div><!-- fragment --><p>The following code contains the description of the below-listed steps:</p><ol type="1">
<li>instantiate PyTorch model</li>
<li>convert PyTorch model into <code>.onnx</code></li>
</ol>
<div class="fragment"><div class="line"># initialize PyTorch ResNet-50 model</div><div class="line">original_model = models.resnet50(pretrained=True)</div><div class="line"></div><div class="line"># get the path to the converted into ONNX PyTorch model</div><div class="line">full_model_path = get_pytorch_onnx_model(original_model)</div><div class="line">print(&quot;PyTorch ResNet-50 model was successfully converted: &quot;, full_model_path)</div></div><!-- fragment --><p><code>get_pytorch_onnx_model(original_model)</code> function is based on <code>torch.onnx.export(...)</code> call:</p>
<div class="fragment"><div class="line"># define the directory for further converted model save</div><div class="line">onnx_model_path = &quot;models&quot;</div><div class="line"># define the name of further converted model</div><div class="line">onnx_model_name = &quot;resnet50.onnx&quot;</div><div class="line"></div><div class="line"># create directory for further converted model</div><div class="line">os.makedirs(onnx_model_path, exist_ok=True)</div><div class="line"></div><div class="line"># get full path to the converted model</div><div class="line">full_model_path = os.path.join(onnx_model_path, onnx_model_name)</div><div class="line"></div><div class="line"># generate model input</div><div class="line">generated_input = Variable(</div><div class="line">    torch.randn(1, 3, 224, 224)</div><div class="line">)</div><div class="line"></div><div class="line"># model export into ONNX format</div><div class="line">torch.onnx.export(</div><div class="line">    original_model,</div><div class="line">    generated_input,</div><div class="line">    full_model_path,</div><div class="line">    verbose=True,</div><div class="line">    input_names=[&quot;input&quot;],</div><div class="line">    output_names=[&quot;output&quot;],</div><div class="line">    opset_version=11</div><div class="line">)</div></div><!-- fragment --><p>After the successful execution of the above code we will get the following output:</p>
<div class="fragment"><div class="line">PyTorch ResNet-50 model was successfully converted: models/resnet50.onnx</div></div><!-- fragment --><p>The proposed in <code>dnn/samples</code> module <code>dnn_model_runner</code> allows us to reproduce the above conversion steps for the following PyTorch classification models:</p><ul>
<li>alexnet</li>
<li>vgg11</li>
<li>vgg13</li>
<li>vgg16</li>
<li>vgg19</li>
<li>resnet18</li>
<li>resnet34</li>
<li>resnet50</li>
<li>resnet101</li>
<li>resnet152</li>
<li>squeezenet1_0</li>
<li>squeezenet1_1</li>
<li>resnext50_32x4d</li>
<li>resnext101_32x8d</li>
<li>wide_resnet50_2</li>
<li>wide_resnet101_2</li>
</ul>
<p>To obtain the converted model, the following line should be executed:</p>
<div class="fragment"><div class="line">python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name &lt;pytorch_cls_model_name&gt; --evaluate False</div></div><!-- fragment --><p>For the ResNet-50 case the below line should be run:</p>
<div class="fragment"><div class="line">python -m dnn_model_runner.dnn_conversion.pytorch.classification.py_to_py_cls --model_name resnet50 --evaluate False</div></div><!-- fragment --><p>The default root directory for the converted model storage is defined in module <code>CommonConfig</code>:</p>
<div class="fragment"><div class="line">@dataclass</div><div class="line">class CommonConfig:</div><div class="line">    output_data_root_dir: str = &quot;dnn_model_runner/dnn_conversion&quot;</div></div><!-- fragment --><p>Thus, the converted ResNet-50 will be saved in <code>dnn_model_runner/dnn_conversion/models</code>.</p>
<h3>Inference Pipeline</h3>
<p>Now we can use <code>models/resnet50.onnx</code> for the inference pipeline using OpenCV C/C++ API. The implemented pipeline can be found in <a href="https://github.com/opencv/opencv/blob/master/samples/dnn/classification.cpp">samples/dnn/classification.cpp</a>. After the build of samples (<code>BUILD_EXAMPLES</code> flag value should be <code>ON</code>), the appropriate <code>example_dnn_classification</code> executable file will be provided.</p>
<p>To provide model inference we will use the below <a href="https://www.pexels.com/photo/brown-squirrel-eating-1564292">squirrel photo</a> (under <a href="https://www.pexels.com/terms-of-service/">CC0</a> license) corresponding to ImageNet class ID 335: </p><div class="fragment"><div class="line">fox squirrel, eastern fox squirrel, Sciurus niger</div></div><!-- fragment --><div class="image">
<img src="../../squirrel_cls.jpg" alt="squirrel_cls.jpg"/>
<div class="caption">
Classification model input image</div></div>
<p> For the label decoding of the obtained prediction, we also need <code>imagenet_classes.txt</code> file, which contains the full list of the ImageNet classes.</p>
<p>In this tutorial we will run the inference process for the converted PyTorch ResNet-50 model from the build (<code>samples/build</code>) directory:</p>
<div class="fragment"><div class="line">./dnn/example_dnn_classification --model=../dnn/models/resnet50.onnx --input=../data/squirrel_cls.jpg --width=224 --height=224 --rgb=true --scale=&quot;0.003921569&quot; --mean=&quot;123.675 116.28 103.53&quot; --std=&quot;0.229 0.224 0.225&quot; --crop=true --initial_width=256 --initial_height=256 --classes=../data/dnn/classification_classes_ILSVRC2012.txt</div></div><!-- fragment --><p>Let's explore <code>classification.cpp</code> key points step by step:</p>
<ol type="1">
<li>read the model with <a class="el" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422" title="Read deep learning network represented in one of the supported formats. ">cv::dnn::readNet</a>, initialize the network:</li>
</ol>
<div class="fragment"><div class="line">Net net = <a class="code" href="../../d6/d0f/group__dnn.html#ga3b34fe7a29494a6a4295c169a7d32422">readNet</a>(model, config, framework);</div></div><!-- fragment --><p>The <code>model</code> parameter value is taken from <code>--model</code> key. In our case, it is <code>resnet50.onnx</code>.</p>
<ul>
<li>preprocess input image:</li>
</ul>
<div class="fragment"><div class="line"><span class="keywordflow">if</span> (rszWidth != 0 &amp;&amp; rszHeight != 0)</div><div class="line">{</div><div class="line">    <a class="code" href="../../d5/df1/group__imgproc__hal__functions.html#ga2fe39d2201b12e1b961ca56b2aff9ff2">resize</a>(frame, frame, <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(rszWidth, rszHeight));</div><div class="line">}</div><div class="line"></div><div class="line"><span class="comment">// Create a 4D blob from a frame</span></div><div class="line"><a class="code" href="../../d6/d0f/group__dnn.html#ga29f34df9376379a603acd8df581ac8d7">blobFromImage</a>(frame, blob, scale, <a class="code" href="../../dc/d84/group__core__basic.html#ga346f563897249351a34549137c8532a0">Size</a>(inpWidth, inpHeight), mean, swapRB, crop);</div><div class="line"></div><div class="line"><span class="comment">// Check std values.</span></div><div class="line"><span class="keywordflow">if</span> (std.val[0] != 0.0 &amp;&amp; std.val[1] != 0.0 &amp;&amp; std.val[2] != 0.0)</div><div class="line">{</div><div class="line">    <span class="comment">// Divide blob by std.</span></div><div class="line">    <a class="code" href="../../d2/de8/group__core__array.html#ga6db555d30115642fedae0cda05604874">divide</a>(blob, std, blob);</div><div class="line">}</div></div><!-- fragment --><p>In this step we use <a class="el" href="../../d6/d0f/group__dnn.html#ga29f34df9376379a603acd8df581ac8d7" title="Creates 4-dimensional blob from image. Optionally resizes and crops image from center, subtract mean values, scales values by scalefactor, swap Blue and Red channels. ">cv::dnn::blobFromImage</a> function to prepare model input. We set <code>Size(rszWidth, rszHeight)</code> with <code>--initial_width=256 --initial_height=256</code> for the initial image resize as it's described in <a href="https://pytorch.org/hub/pytorch_vision_resnet/">PyTorch ResNet inference pipeline</a>.</p>
<p>It should be noted that firstly in <a class="el" href="../../d6/d0f/group__dnn.html#ga29f34df9376379a603acd8df581ac8d7" title="Creates 4-dimensional blob from image. Optionally resizes and crops image from center, subtract mean values, scales values by scalefactor, swap Blue and Red channels. ">cv::dnn::blobFromImage</a> mean value is subtracted and only then pixel values are multiplied by scale. Thus, we use <code>--mean="123.675 116.28 103.53"</code>, which is equivalent to <code>[0.485, 0.456, 0.406]</code> multiplied by <code>255.0</code> to reproduce the original image preprocessing order for PyTorch classification models:</p>
<div class="fragment"><div class="line">img /= 255.0</div><div class="line">img -= [0.485, 0.456, 0.406]</div><div class="line">img /= [0.229, 0.224, 0.225]</div></div><!-- fragment --><ul>
<li>make forward pass:</li>
</ul>
<div class="fragment"><div class="line">net.setInput(blob);</div><div class="line">Mat prob = net.forward();</div></div><!-- fragment --><ul>
<li>process the prediction:</li>
</ul>
<div class="fragment"><div class="line"><a class="code" href="../../dc/d84/group__core__basic.html#ga1e83eafb2d26b3c93f09e8338bcab192">Point</a> classIdPoint;</div><div class="line"><span class="keywordtype">double</span> confidence;</div><div class="line"><a class="code" href="../../d2/de8/group__core__array.html#gab473bf2eb6d14ff97e89b355dac20707">minMaxLoc</a>(prob.reshape(1, 1), 0, &amp;confidence, 0, &amp;classIdPoint);</div><div class="line"><span class="keywordtype">int</span> classId = classIdPoint.x;</div></div><!-- fragment --><p>Here we choose the most likely object class. The <code>classId</code> result for our case is 335 - fox squirrel, eastern fox squirrel, Sciurus niger:</p>
<div class="image">
<img src="../../opencv_resnet50_test_res_c.jpg" alt="opencv_resnet50_test_res_c.jpg"/>
<div class="caption">
ResNet50 OpenCV C++ inference output</div></div>
</div></div><!-- contents -->
<!-- HTML footer for doxygen 1.8.6-->
<!-- start footer part -->
<hr class="footer"/><address class="footer"><small>
Generated on Fri Apr 2 2021 11:36:34 for OpenCV by &#160;<a href="http://www.doxygen.org/index.html">
<img class="footer" src="../../doxygen.png" alt="doxygen"/>
</a> 1.8.13
</small></address>
<script type="text/javascript">
//<![CDATA[
addTutorialsButtons();
//]]>
</script>
</body>
</html>
